Suppression of vascular structures in images

ABSTRACT

Image processing techniques may include a methodology for normalizing medical image and/or voxel data captured under different acquisition protocols and a methodology for suppressing selected anatomical structures from medical image and/or voxel data, which may result in improved detection and/or improved rendering of other anatomical structures. The technology presented here may be used, e.g., for improved nodule detection within computed tomography (CT) scans. While presented here in the context of nodules within the lungs, these techniques may be applicable in other contexts with little modification, for example, the detection of masses and/or microcalcifications in full field mammography or breast tomosynthesis based on the suppression of glandular structures, parenchymal and vascular structures in the breast.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a non-provisional patent application derivingpriority from U.S. Provisional Patent Application No. 61/971,042, filedon Mar. 27, 2014, and incorporated by reference herein.

FIELD OF ENDEAVOR

The present disclosure may relate generally to suppressing unwantedstructures within a medical image and/or voxel data using featureextraction and successive model-based prediction methods and may furtherrelate to techniques that may improve detection of lung nodules withincomputed tomography (CT) scans. Model-based prediction in the context ofthis disclosure is defined as the use of an analytical model(s),empirical model(s) or combination thereof, e.g., neural network, topredict a value (e.g., a pixel or voxel) based on measures, eithercomputed or derived from pixels/voxels.

BACKGROUND

It is widely recognized that object detection is a challenging problem.There are many aspects that make object detection difficult for computervision systems, including factors such as variations in imageacquisition, complexity of object appearance, and significantvariability in object backgrounds (usually referred to as clutter), toname just a few. In the domain of medical imaging, an “object” mightrefer to a particular component of normal anatomy, the location of anon-anatomical object, or the presence of disease such as a tumor.

One important application of object detection in medical imaging is thedetection of lung nodules, or masses, in CT scans of the chest. Despitemore than two decades of effort, the general problem of machine noduledetection remains unsolved, and human detection remains limited. Weargue that a significant reason for this is a failure to address onesignificant component of what makes the problem difficult: the complexinteraction of nodules with pulmonary vessels, and the variation inappearance due to varying acquisition protocols.

SUMMARY OF THE DISCLOSURE

Various aspects of this disclosure may include an approach fornormalizing medical image and/or voxel data captured under differentacquisition protocols, and/or a method for suppressing selectednon-nodule structures within medical image and/or voxel data of thechest. Most non-nodule structure is vascular content, and therefore, theterm “vessel suppression” will be used in this disclosure as a generalterm for such non-nodule structure suppression. However, the disclosedtechniques may also apply to structures other than “vessels”/vascularstructures, e.g., bronchial walls and fissure lines in the thorax, aswell as, on occasion, man-made objects that take on tubular-likeproperties. This may further extend to other body parts (e.g., thebreast, the heart, the head), other modalities (e.g., ultrasound,tomosynthesis, etc.) and/or other domains (e.g., video surveillance,military targeting). The techniques may be used for the purposes ofimproved nodule detection, nodule characterization and/or improvedrendering of selected anatomically suppressed or enhanced image data.While the techniques are described herein with specific reference tonodules within the lungs, similar methodologies may be applied in othercontexts.

Additional features and advantages of various aspects of this disclosurewill be apparent from the detailed description that follows, taken inconjunction with the accompanying drawings, which together illustrate,by way of example, features of various aspects of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

In the accompanying drawings:

FIG. 1 shows an example of a process diagram, e.g., for CT acquisitionnormalization, according to an aspect of the disclosure;

FIG. 2 shows a further process diagram, showing further processingaccording to various aspects of the disclosure;

FIGS. 3A and 3B show an example of a slice of a volume before and afternodule simulation; FIG. 3C shows the same slice shown in FIG. 3B aftervessel suppression;

FIG. 4 shows a process diagram of an example of a prediction phase,according to an aspect of the disclosure;

FIG. 5 provides a conceptual depiction of tissue separation, accordingto an aspect of the disclosure;

FIGS. 6A-6C show an image example of a ground glass nodule with vesselsremoved; and

FIG. 7 shows a conceptual diagram of a system in which various aspectsof the disclosure may be implemented.

DETAILED DESCRIPTION

There are several approaches that may be taken to nodule detection,which may include template matching (e.g., Q. Li, S. Katsuragawa, and K.Doi, “Computer-aided diagnostic scheme for lung nodule detection indigital chest radiographs by use of a multiple-template matchingtechnique,” Medical Physics, 2001, 28(10): 2070-2076; hereinafter “Li etal. 2001”), multi-level thresholding (e.g., S. Armato, M. Giger, and H.MacMahon, “Automated detection of lung nodules in CT scans: preliminaryresults,” Medical Physics, 2001, 28, 1552-1561; hereinafter “Armato etal. 2001”), enhancement filters (e.g., A. Frangi, W. Niessen, K.Vincken, and M. Viergever, “Multiscale vessel enhancementfiltering,”MICCAI, 1998, 130-137; hereinafter “Frangi et al. 1998;” andQ. Li, S. Sone. and K. Doi, “Selective enhancement filters for nodules,vessels, and airway walls in two- and three-dimensional CT scans,”Medical Physics 2003, 30(8): 2040-2051; hereinafter “Li et al. 2003”),and voxel classification (e.g., D. Wu, L. Lu, J. Bi, Y. Shinagawa, K.Boyer, A. Krishnan, and M. Salganicoff, “Stratified learning of localanatomical context for lung nodules in CT images,” CVPR, 2000,2791-2798.hereinafter “Wu et al. 2000”).

Template matching (e.g., as in Li et al. 2001) may involve measuring thesimilarity about each voxel to a set of templates. The more varied theobject appearance, the more templates may be needed for reasonableperformance. This scaling behavior may make template matchinginefficient in difficult domains.

Multi-level thresholding may be used in CT (e.g., as in Armato et al.2001), in part because voxel values, which may be defined in Hounsfieldunits (HU), may have a meaningful interpretation. Knowledge of noduledensity, for example, may be used for setting the thresholds. Thistechnique may encounter some difficulties, one of which may be propermeasurement of object morphology for a given threshold. Nodules may beconnected to surrounding structures, which may make accuratemorphological assessment difficult. To compensate, such approaches mayembed a morphological post-processing step to remove connectedstructures. The morphological post-processing may, however, alter thenodule to such an extent that it may become undetectable. One mightadapt this process by using more elaborate post-processing, e.g.,rule-driven adaptation, but such rule driven adaption may lead tobrittleness, and the method may begin to look more like templatematching.

Filter enhancement methods (e.g., as in Frangi et al. 1998 and Li et al.2003) may improve upon template matching by being adaptive to localstructures. One such approach may be to estimate local structure derivedfrom local tensor information. Two example tensors are the 3×3 Hessianmatrix and the 3×3 structure tensor, where the “3” refers to the numberof spatial dimensions. The eigenvalues from these tensors may be used toquantify the degree of “tubeness”, “blobness” or “plateness” at eachvoxel. These indicators may be combined to derive a composite featureindex. While such analysis may be simple and analytically neat, it mayencounter some limitations. First, the expressions used to combine theinformation may be based on idealizations of nodules and vessels thatmay not be true in reality. For example, nodules are not generallyperfect spheres, and vessels are not generally perfect cylinders. As themethods may only use information captured by first- and second-orderderivatives, the performance may falter in more complex regions such aswhere vessels bifurcate or where nodules become attached to vascularstructure. Lastly, as the methods are based on idealizations, combiningindicators across multiple scales may not be easy.

Voxel classification, e.g., as in Wu et al. 2000, may involve extractingfeatures from a CT scan that may then be used by a classification methodto produce probabilities, or other outputs, that may indicate if anodule is present or not. Voxel classification may need large amounts ofmanually labeled data, which may be impractical. The voxelclassification method may also suffer from sample bias, which means thatit may be specifically tuned to the types of nodules collected to trainit, which may result in missed nodules.

Another method is found in R. Wiemker, T. Buelow and T Klinder, “VisualSuppression of Selective Tissue in Image Data,” U.S. Patent ApplicationPublication No. 2015/0063669 (hereinafter “Wiemker et al. 2015”). InWiemker et al., the inventors describe a method for suppressing vascularstructure as a weighted combination of original data with suppresseddata. This type of methodology may often be referred to as “inpainting”in the literature. One such approach to object removal may be pyramidblending. The weight used for blending may be derived from a locallikelihood of “vesselness,” which may correspond to a value between zeroand one. How the measure is actually derived is never described inWiemker et al. 2015; only its use as a means to blend image data withand without suppression is described. Wiemker et al. describes variousmodes of modification of the likelihood for emphasizing or deemphasizingthe weight. The mechanism for getting the “opacity mappings” of Wiemkeret al. is less clear. The description seems to imply that the density ofvoxels is decreased by a type of look-up table.

It has also been recognized that it may be useful to develop automatedmethods for nodule detection and segmentation, but which may alsoprovide adjunctive information that may be in the form, e.g., ofsecondary volumes that can be used by human experts (see, e.g., B. VanGinneken, S. Armato, et al., “Comparing and combining algorithms forcomputer-aided detection of pulmonary nodules in computed tomographyscans: The ANODE09 study,” Med. Image Analysis, 2010, 14(6), 707-722,and E. M. Van Rikxoort, B. Van Ginneken, “Automated segmentation ofpulmonary structures in thoracic computed tomography scans: a review,”Physics in Medicine and Biology, 2013, 58, 187-220).

Various aspects of the present disclosure may be related to frameworksbuilt by the present inventors for normal anatomy suppression (see,e.g., U.S. Patent Application Publication No. 2009/0290779 to Knapp etal. (hereinafter, “Knapp et al. 2009”) and U.S. Patent ApplicationPublication No. 2013/0223711 to Knapp et al. (hereinafter, “Knapp et al.2013”), both of which are incorporated by reference herein). In Knapp etal. 2009), models were built by predicting an alternative image wherethe density of bones is removed. In Knapp et al. 2013, a pectoral musclesuppression technique may predict image data where the bias associatedwith the pectoral muscle is removed.

In the context of the present disclosure, use of the term “suppression”implies that anatomical structures (such as vessels) are actuallyremoved from an image and are not simply made less dense. Aspects of thepresent disclosure may relate to building a prediction model that may“predict out” undesired density, as will be described further below.

Reference will now be made to the exemplary embodiments illustrated inthe drawings, and specific language will be used herein to describe thesame. It will nevertheless be understood that no limitation of the scopeof the invention is thereby intended. Alterations and furthermodifications of the inventive features illustrated herein, andadditional applications of the principles of the inventions asillustrated herein, which would occur to one skilled in the relevant artand having possession of this disclosure, are to be considered withinthe scope of the invention.

In identifying nodules in a CT scan, one may ideally like to take the CTscan and to suppress structures other than nodules. To start thisprocess each scan may be normalized to account for variations associatedwith acquisition. This may enhance robustness and simplify furtherprocesses. The main steps in this normalization process, according to anaspect of this disclosure, may be seen in FIG. 1. The normalizationprocess may begin with “body segmentation” 10, in which a patient's bodymay be segmented from other structures within the field-of-view of theinput CT image, and may subsequently segment the air regions associatedwith the patient's respiratory system. Following this, the air regionmay be calibrated 11 to a fixed value so that the lung density remainsnear a fixed value. After density calibration 11, the CT scan's noiseproperties may be analyzed, and an adaptive local averaging method maybe used in order to suppress noise artifacts 12. Following noisesuppression 12, the image may be processed so that the contrast detailis as consistent as possible from one scan to the next 13. This may beachieved, e.g., using techniques similar to those used in Knapp et al.2009 and U.S. Patent Application Publication No. 2010/0266189 to Knappet al. (hereinafter, “Knapp et al. 2010;” Knapp et al. 2010 is alsoincorporated by reference herein), in which histogram matching was usedon a multi-scale representation in order to calibrate contrast detail.Lastly, the volume may be resized (resampled to a specified size, e.g.,in millimeters), which may be performed in-plane (within a CT image)and/or out of plane (across CT images), which may involve usinginterpolation, so that slice spacing and thickness may be made asconsistent as possible from one scan to the next 14; this may, again,use techniques described in Knapp et al. 2009 and/or Knapp et al. 2010.

In order to perform vessel suppression, given a volume, one may wish togenerate a volume with the vessels predicted out, while being carefulnot to remove other structures such as nodules. This may be achieved bya strategy of “forward through simulation, inversion with prediction,”which will be clarified in the subsequent discussion.

FIG. 2 shows an example of an overall process flow according to anaspect of this disclosure. The process may begin with case selection,i.e., a set number of CT volumes of the thorax that capture arepresentative amount of normal (anticipated) variability. Using theseselected cases, which may have been normalized for acquisitionvariation, a vessel suppressed volume may be generated for the purposeof model construction, shown in FIG. 2 as target formation 20. In onetechnique that may be used to create a vessel-suppressed volume (towhich the invention is not limited), one may first compute a localminimum intensity projection (min-ip) along the axial direction of theCT volume. This min-ip operation may serve to suppress all, orsubstantially all, content within the lung fields. The min-ip volume maybe blended with the CT volume using a slightly smoothed version of thesegmented vessel mask, or in order to suppress all structures, mayblended with the CT volume with a mask of known nodule locations. Thevessel and/or nodule masks may be generated using an automated algorithmor a semi-automated method; or they may be derived from manual outlines.As these masks are to be used solely for the offline process of creatingtarget data, the actual mechanism used to create them is not essential;however, the more accurate they are, the better. In order to have asufficient number of examples of nodules interacting with pulmonaryvessels, one may use nodule simulation, e.g., as in Knapp et al. 2009.That is, synthetic nodules may be inserted into the unsuppressed and/orvessel-suppressed volumes. The result may be a pair of volumes with andwithout vessels, but both with unaltered, or substantially unaltered,nodules. FIG. 3A is a slice from a represented example, while in FIG. 3Bwe see the same slice, only with simulated nodules added. In FIG. 3C wesee the same slice from FIG. 3B, only with vessels suppressed, and inwhich the nodules have been left unaltered.

Once the pairs of input-target volumes are created 20, one may processeach image by passing it through a feature generation process 21. Thisprocess may be used to extract voxel-level features, where an exampleset of features may be as follows:

-   -   Multi-scale Gaussian derivatives with a range of orders;    -   Local minimum and maximum intensity projections;    -   In-plane features computed from and/or on the local minimum and        maximum intensity projections.

Other features may be derived based on location or the derivativefeatures themselves (shape indices, curvature, etc . . . ) or learnedusing model-based approaches, such as patch analysis or deep neuralnetworks. Given the large collection of data, a model, or a set ofmodels, may be generated 22. According to one aspect of this disclosure,multi-layer feed-forward neural networks may be used for this purpose.These models may subsequently be able to predict suppressed data,without the use of a segmentation mask or object indicators, similar tomethods found in Knapp et al. 2013. Once a model or set of models havebeen built, their performance on validation data can be assessed 23.This may lead to the selection of a particular subset of models and/orcould be used to further guide the training process by the selection ofadditional cases for training or the creation of more simulated nodules.

FIG. 4 depicts an example of a high-level flow diagram from a normalizedCT scan to a suppressed scan, according to aspects of this disclosure.The extracted features 40 may be the same as those used in modelgeneration 22. The prediction phase 41 may involve applying a set ofprediction models, e.g., neural networks, whose outputs may be combined.The combining may be performed, e.g., by averaging, by using a costfunction to select an optimal one of the outputs, or both (i.e.,selecting an optimal set of outputs to average). In this aspect of thedisclosure, multiple model outputs may be obtained for multiple imagezones, where each image zone may be identified by its anatomicallocation and/or by its pixel density. Post-processing 42 may be used toadjust for the normalization process so that the suppressed volume isin-line with the original image data, if desired for display purposes,as it may not be needed for computer-aided detection.

FIG. 5 provides a conceptual example of how the overall technique mayoperate on a real nodule and may generate two representations. An image50 may undergo anatomy suppression 51, to result in two representations52. The upper representation in 52 may be a suppressed volume, which mayallow for easier detection and segmentation of nodules; and the lowerrepresentation in 52 may be a vessel volume, which may simply be thedifference between the CT data and the suppressed data. The vesselvolume may be useful, e.g., for vessel segmentation or registration withprior scans.

FIG. 6, consisting of FIGS. 6A-6C is an up-close view of an example of areal ground glass nodule with apparent vessel superposition. FIG. 6Ashows the CT image. FIG. 6B shows an up-close view of the ground glassnodule image, and FIG. 6C shows a vessel-suppressed image. As can beseen, the algorithm may remove the vascular structure while preservingthe nodule content.

Various embodiments of the invention may comprise hardware, software,and/or firmware. FIG. 7 shows an exemplary system that may be used toimplement various forms and/or portions of embodiments according tovarious aspects of this disclosure. Such a computing system may includeone or more processors 72, which may be coupled to one or more systemmemories 71. Such system memory 71 may include, for example, RAM, ROM,or other such machine-readable media, and system memory 71 may be usedto incorporate, for example, a basic I/O system (BIOS), operatingsystem, instructions for execution by processor 72, etc. The system mayalso include further memory 73, such as additional RAM, ROM, hard diskdrives, or other processor-readable media. Processor 72 may also becoupled to at least one input/output (I/O) interface 74. I/O interface74 may include one or more user interfaces, as well as readers forvarious types of storage media and/or connections to one or morecommunication networks (e.g., communication interfaces and/or modems),from which, for example, software code may be obtained or provided(e.g., by downloading or uploading).

It is to be understood that the above-referenced arrangements are onlyillustrative of the application for the principles of the presentinvention. Numerous modifications and alternative arrangements can bedevised as described in the usage and extension to other applicationsand domains sections without departing from the spirit and scope of thepresent invention.

To elaborate, generation of a vessel-suppressed volume may have manyuses outside nodule detection and characterization. These may include:

-   -   Simpler and more robust segmentation of vessels, which can be        used as inputs to other processes, such as pulmonary embolism        detection;    -   Improved vessel tree navigation to biopsy nodules attached to        the vessels based on improved visualization of the vascular        tree;    -   Improved detection of disease related to the vascular structure        that leads the thickening and/or constriction of the vessels;    -   Use of the vessel volume, or segmented mask, as an input to a        scan registration process, thus allowing for assessment of any        potential abnormal changes, such as nodule growth;    -   Use of the suppressed volume to generate a more-informative        display, such as the generation of a maximum intensity        projection of the lung (which may, in effect, provide a        high-level indication of where nodules may be located);    -   Two-dimensional reconstructions with both bones and vessels        suppressed.

Furthermore, the present techniques may be applied in otherapplications/domains. Many extensions of the framework presented abovego beyond vessel suppression, and the following is merely a list (whichis not intended to be exhaustive) of some further applications:

-   -   Vessel suppression in the brain for highlighting cerebral        aneurysms;    -   Vessel suppression in CT liver scans for highlighting nodules        and improving the ability to segment the vascular structure,        perhaps for registration purposes;    -   Vessel suppression in fundus images of the eye for highlighting        micro-aneurysms and other diseases;    -   Vessel suppression in the breasts (volumes or images acquired        via tomosynthesis or mammograms, respectively) for detection or        removal of benign calcification, for use in registration or        other processes.

While the present invention has been shown in the drawings and fullydescribed above with particularity and detail in connection with what ispresently deemed to be the most practical and preferred embodiment(s) ofthe invention, it will be apparent to those of ordinary skill in the artthat numerous modifications can be made without departing from theprinciples and concepts of the invention as set forth herein.

What is claimed is:
 1. A method of obtaining one or more imagecomponents from data representing one or more images or image volumes,the method comprising: normalizing and pre-processing the data to obtainprocessed data; extracting features from the processed data to obtain aset of extracted features; and performing model-based prediction usingat least one model based on the set of extracted features to predict oneor more components based on a set of target data.
 2. The method of claim1, further comprising: obtaining a prediction output with one or more ofthe components removed.
 3. The method of claim 1, further comprising:subtracting one or more components predicted by the model-basedprediction from the data to obtain data with the one or more componentsremoved.
 4. The method of claim 1, wherein the data comprises aradiographic CT series, and wherein one or more components comprise onlyvascular components;
 5. The method of claim 1, wherein the datacomprises a radiographic CT series, and wherein one or more of thecomponents comprise only nodular structures.
 6. The method of claim 1,further comprising inserting simulated nodules, measured nodules, orboth into original volumes, anatomy-suppressed volumes, or both, priorto using the original volumes, anatomy-suppressed volumes, or both forgenerating the target data.
 7. The method of claim 1, wherein saidnormalizing and pre-processing includes: performing noise suppression toobtain noise-suppressed data; and performing a bandpass decomposition onthe noise-suppressed data.
 8. The method of claim 6, wherein saidnormalizing and pre-processing further comprises: performing at leastone operation on at least one result of said bandpass decomposition,wherein the at least one operation is selected from the group consistingof: gray scale registration and enhancement:
 9. The method of claim 6,wherein said normalizing and pre-processing includes: data resizing toobtain target resized data.
 10. The method of claim 1, wherein saidextracting features includes: obtaining Gaussian derivatives acrossdifferent scales.
 11. The method of claim 1, wherein said performingmodel-based prediction comprises: applying multiple prediction models toobtain multiple suppressed predictions for one or more volume voxels;and combining the multiple suppressed predictions to obtain a combinedestimate.
 12. The method of claim 11, wherein said combining themultiple suppressed predictions comprises at least one of averaging themultiple suppressed predictions or selecting an optimal suppressedprediction based on a specified cost function.
 13. The method of claim1, wherein said performing model-based prediction comprises: applyingmultiple prediction models corresponding to multiple image zones toobtain predictions for the pixels/voxels of the multiple image zones,wherein the image zones are defined based on anatomical location, onvoxel density values, or on both.
 14. The method of claim 1, furthercomprising: downloading software instructions to implement saidnormalizing and pre-processing, said extracting features, and saidperforming model-based prediction.
 15. A machine-readable storage mediumcontaining instructions designed to implement operations comprising:normalizing and pre-processing the data to obtain processed data;extracting features from the processed data to obtain a set of extractedfeatures; and performing model-based prediction using at least one modelbased on the set of extracted features to predict one or more componentsbased on a set of target data.
 16. The medium of claim 15, wherein theoperations further comprise: obtaining a prediction output with one ormore components removed.
 17. The medium of claim 15, wherein theoperations further comprise: subtracting one or more components from thedata to obtain data with the one or more components removed.
 18. Themethod of claim 1, wherein the one or more images or image volumescomprise a radiographic image, and wherein the one or more componentscomprise one or more structures normally found in such a radiographicimage.
 19. A method of constructing an alternative projection comprisingusing a anatomical suppressed volume obtained by the method of claim 1.20. The method of claim 20, further comprising displaying thealternative projection.
 21. The method of claim 20, further comprisingperforming disease detection, segmentation, or registration of theanatomical suppressed volume with a different volume.
 22. The method ofclaim 20, further comprising using the alternative projection toregister the data with second data from a different modality from amodality used to obtain the data.
 23. An apparatus comprising: at leastone processor; and the computer-readable medium according to claim 15.